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Evaluation Packet Overview 



Mediation has rapidly become a useful approach for addressing concerns on college and 
university campuses across the country. It not only is an alternative to contentious and litigious 
dispute resolution methods, it also teaches valuable communication, listening, and problem 
solving skills. The process of mediation holds great promise for assisting students, staff, faculty, 
and others in developing appropriate solutions to a wide range of concerns. Do you wish to start 
or expand a conflict resolution program on your campus? Do you know if the service is needed? 
Who should you target? If you already have a program in place do you know whether things are 
being implemented as planned? How many people do you serve with your conflict resolution 
program? Are people satisfied with the results? 

This packet is intended to address these types of questions and concerns. The materials included 
in this packet were developed by a group of researchers and campus mediation program staff and 
volunteers that gathered together at Wayne State University in Detroit, Michigan, from April 4-6, 
2001. While we came from a variety of institutions of higher education - large and small, public 
and private - our shared aim was twofold: (1) to improve the quality of campus conflict 
resolution program evaluation efforts, and; (2) to increase the frequency of these efforts. Rather 
than develop a standard, fixed set of materials to be used by everyone, the group chose to 
develop a set of tools and approaches that can be adapted to meet the specific local needs of 
programs at various stages of development. To this end, the materials have been divided into 
four (4) modules, each of which provides insight into and guidance for assessing a different 
aspect of program development. These include: 



• Module I: 

• Module II: 

• Module III: 

• Module IV : 



Needs Assessment 
Process Monitoring 
Evaluation of Training 
Mediation Outcomes and Impact 



This evaluation packet is part of a larger effort to establish a web-based resource for campus- 
based conflict resolution service programs. The Campus Conflict Resolution Resources website 
(www.campus-adr.org) provides information and links to, for example, bibliographic references, 
funding sources, mediator training, and conferences, in addition to resources for evaluation 
efforts. 



We need your help. Your feedback on the usefulness of these materials is critical to their success 
(add weblink for feedback form) . We also need samples of data collection instruments (e.g., 
surveys, interview questions) and of evaluation reports to include on the Resources website. 

We also hope you will share some of the results of your evaluation efforts, either by posting new 
or modified data collection instruments you have developed, or some results from your local 
efforts to study and evaluate campus conflict resolution processes and services. Look for more 
information on how to get involved by visiting the website ( www.campus-adr.org) , click on the 
Program Evaluation Office found in our online Conflict Resolution Services Building. 

A note about terminology - Throughout this packet we use the terms conflict resolution and 
mediation interchangeably. We also use program and sendee interchangeably when referring to 
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conflict-related efforts. Lastly, although we sometimes only mention college or university 
campuses, these materials are appropriate for use by programs on all institutions of higher 
education. 



Why Do Evaluation Research? 

Research is not often a high priority for people engaged in the exciting, challenging, and 
sometimes tiring work of setting up and maintaining a campus conflict resolution service. The 
very term, research, conjures up images of the scientist in the laboratory forever worrying over 
some obscure concoction. Nevertheless, research, particularly evaluation research, is an 
important key to long-term program success. The still emergent field of conflict resolution in 
higher education will only truly mature to the extent that we begin to study and share broadly 
what works, what does not work, and why. Programs that find ways to include research as a 
basic part of what they do are in a very good position to help move both themselves, and the field 
of conflict resolution, forward. 

Evaluation research can also be considered one of the ethical responsibilities of competent 
mediation service providers. The Model Standards of Conduct for Mediators, jointly developed 
by the American Arbitration Association, the American Bar Association, and the Society for 
Professionals in Dispute Resolution in 1995, addresses this issue directly. One of the standards, 
Obligations to the Mediation Process , indicates: "mediators have a duty to improve the practice 
of mediation." This clearly implies a responsibility on the part of both individuals and programs 
to evaluate and refine the practice of dispute resolution. 

In general, evaluation research can serve quite a number of related purposes: 

Justifying and explaining the program - Providing credible proof to skeptics can 
help increase the sustainability of programs. Evaluation data are often useful for 
justifying, defending, or explaining a program. Sponsors may wish to know details 
regarding caseloads, types of conflicts, and outcomes. 

Program planning and decision-making - Research can help determine how best to 
allocate time and resources now and in the future. Data can also be used to refine 
public relations or outreach strategies or to make decisions about training needs. 
Questions regarding duplication of services can also be explored. 

Improving services - Research can help practitioners understand what is working and 
what is not and identify larger patterns that may go unnoticed day to day. For instance, 
systematic follow-up contact with disputants might reveal that the compliance rate for 
certain kinds of agreements goes down a few months after mediation. This finding 
could suggest needed changes in mediation procedures. 

Addressing a specific problem area - Having good data on areas of concern can help 
the mediation program respond more effectively. For instance, the director may have a 
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concern about the limited use of the service by diverse cultural groups. Part of the 
monitoring might include recording the ethnicity of parties using the service. 

Assessing mediator needs and impact - Evaluations can be used to assess the needs, 
concerns, and effectiveness of mediators. Questions relating to the effectiveness of the 
training, ongoing mediator professional development needs, and disputant satisfaction 
with mediators assigned to them or the process used can be quite helpful. 

Identifying, at the start of a project, the purposes likely to be most important will increase the 
chances of gathering data that will be both credible and a good match for program needs. It also 
provides the opportunity to develop standardized forms and practices that will endure as the 
program matures and office personnel change. 

Types of Evaluation Research 

There are two broad categories of evaluation research: formative and summative. Formative 
evaluation, also known as program or process monitoring, is used to keep track of how a 
program is being implemented, assess whether a project is going according to plan, document 
problems or new possibilities arising as the program grows, and provide useful strategic 
information for program developers. Summative evaluation, also referred to as outcome 
evaluation, examines the outcomes of the program to see if they meet expectations. It focuses on 
determining whether the program is meeting its stated objectives, whether it is worth continuing 
or expanding, and how effective it is. Most programs end up doing some combination of the 
two. It is preferable to gather formative information as the program develops, making 
adjustments as necessary, rather than proceed without gathering data until it becomes time to 
answer critical questions of continued fundability. 

Information Gathering 

There are many potential sources of useful information, some covered in this packet, and others 
left to your ingenuity and imagination. Much of the information for formative evaluation comes 
from office forms (often referred to as monitoring records) that are set up and maintained by staff 
and volunteers. These include, but are not limited to, telephone and office contact logs, case 
intake forms, and public relations activity logs that track events and the number of participants 
involved. Organizing the data collected by a monitoring system can be accomplished in any 
number of ways depending on data processing capabilities and needs. In small programs, a 
simple paper filing system and periodic (monthly or by semester) manual tabulation procedures 
might be adequate. In a larger program, data might be converted for analysis on a personal 
computer, perhaps using a mediation case management software package or a general statistical 
or graphing program. 

In addition to data from monitoring records, programs may develop questionnaires, surveys, 
focus groups, and observation or interview guides to gather other kinds of information deemed 
useful for evaluation purposes. Also, when permissible and available, secondary or archival data 
analysis of records prepared by related programs or services may be valuable. For instance, it 
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can be useful to note changes in complaint or case patterns at other programs or offices that in 
some way relate to the mediation effort. 



Some examples of the kinds of data that program coordinators might be interested in collecting 
and related data gathering tools are listed in Table 1. In each evaluation module in this packet 
we provide suggestions on how and when you might want to collect the relevant data. 

Table 1: Sample Data Gathering Needs and Sources 



Desired Information 


Possible Data Sources 


Patterns of Referral 


Referral forms 

Intake or request-for-mediation forms 
Case files indicating referrals made to other offices 
Post-mediation report forms completed by mediators 
Interviews with referral sources 


Patterns of Program Use 


Telephone and contact logs 

Demographic information gathered on intake forms 
Pre- or post-mediation questionnaires 
Key informant interviews 


Mediators’ Skill Development 


Mediator observation forms (during training & mediations) 
Pre- and post-training questionnaires 
Post-mediation disputant satisfaction surveys 
Post-mediation self or co-mediator evaluation/debriefing forms 


Scope & Extent of Outreach 


Public relations (PR) event log 

Log of number, kind, location of PR materials distributed 
Web site "hit" counters 

"How did you hear about center" questions at intake 
Media clippings file 


Program Efficiency 


Time from first contact to resolution (from Intake form) 

Length of mediation session (from post-mediation report form) 
Percentage of cases reaching agreement (from case files) 

Costs per case - time, money, comparison to other processes 


Disputants' Experiences 


Post-mediation surveys 
Follow-up telephone interviews 


Case & Problem Categories 


Intake forms 
Case files 

Generic case summaries prepared by mediators 
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A note about participation - All forms of data collection should be accessible to any mediation 
participant, so particular care needs to be paid to ensure that materials and methods are available 
in a variety of formats. To ensure that evaluation is conducted appropriately and respectfully, 
interviews and questionnaires should be reviewed for accessibility to all possible respondents 
and users of these materials. For example, you many need questionnaires available in large- 
print, Braille, and in languages other than English. For interviews, ensure they are conducted at 
accessible facilities and interviewers are prepared to accommodate a wide range of languages 
and language skills as well as cultural contexts. Regardless of the data collection method, 
sufficient time should be provided to allow for completion of any evaluation material. 

A note about confidentiality - To ensure that participants provide open and honest responses to 
evaluation instruments, it is incumbent on program managers to safeguard the anonymity of the 
individuals and the confidentiality of their responses. Provide assurance to participants on every 
written questionnaire and as part of every structured interview or observation. Prior to 
undertaking any evaluation, agencies must determine which data to collect and how that data will 
be treated. For example, is any of the material open to public scrutiny? If the evaluation process 
calls for follow-up contacts with participants, how will an identifying code be developed and 
where will the list that links the code to specific cases be kept? All policies related to the data 
collection effort should be made available to participants at every stage of the evaluation. 

Using the Evaluation Results 

After the various forms of data have been collected, they need to be analyzed and prepared for 
presentation. Software packages for large amounts of quantitative (SPSS, SAS) and qualitative 
(The Ethnograph, ATFASti, NUD*IST) data are readily available on most campuses for use in 
data analysis. Some specialized survey programs that help streamline data analysis are also 
available. Excel is a basic, yet workable, option as well. 

Once the findings are compiled, strategies for reporting the results need to be developed that will 
appeal to the intended audience (e.g., sponsor, clients, mediators). Several reports may need to 
be developed to provide different kinds and amounts of information depending on the audience. 

If the information will be presented to the media, a press release may be needed as well. Some 
readers like lots of charts and figures; others prefer text. For most audiences, evaluation reports 
should include implications of the findings for: (1) program operation and maintenance; (2) 
expansion, redirection, and sustainability; and (3) recommendations for future short- and long- 
term actions that can be taken to improve the program. 

It can be quite helpful to review a draft of the results of the evaluation with core staff, advisers, 
and volunteers before completing the report. A meeting designed to discuss the findings can be 
held to get input and interpretations. These additional perspectives can strengthen the final 
report and convey respect for the groups most likely to be influenced by the release of the report. 
An evaluation is essentially a device for program staff and volunteers to use to make adjustments 
and improve program services. Taken in this light, a report should not be presented as the final 
word on the program but rather as part of a continuous and evolving process of development and 
refinement. 
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Resources for Mediation Evaluation Research 



A number of useful research tools and guides to program evaluation are available for reference 
purposes. Some are tailored specifically to the mediation context, and others are more general. 

I. Mediation Program Evaluation Guides 

Although none of the following guides focus specifically on the higher education context, they 
may still be of considerable value to individuals designing evaluations of mediation programs. 

A Self-Evaluation Manual for Community-Based Mediation Projects: Tools for Monitoring 
and Recording Data, by T. Roberts. Vancouver, Canada: University of Victoria Institute for 
Dispute Resolution, 1993. This example-filled manual comes out of an evaluation, conducted in 
1990-1991, of two community-based mediation organizations in British Columbia. It provides a 
variety of useful ideas, and sample data monitoring and gathering instruments. Available from: 
http://www.dispute.resolution.uvic.ca or http://www.nicr.ca/ 

Evaluating Agency Alternative Dispute Resolution Programs: A Users' Guide to Data 
Collection and Use, by E. S. Rolph and E. Moller. Santa Monica, California: RAND Institute for 
Civil Justice, 1995. The Administrative Conference of the United States asked the Institute for 
Civil Justice to prepare a manual and develop prototype data collection instruments to assist 
those with responsibility for evaluating federal agency alternative dispute resolution programs. 
The manual discusses issues in designing evaluations, lays out approaches to data collection, 
provides sample data analysis plans, and includes a number of prototype data collection 
instruments. Available from: http://www.rand.org/ 

CADRE Resource Guide: Using Evaluation Data to Improve Quality, by Timothy Hedeen. 
This resource guide presents an overview of mediation evaluation, with a specific focus on how 
to understand and make use of evaluation findings. It is designed to help program managers 
understand evaluation processes and outcomes, rather than serving as a how-to manual on 
evaluation. CADRE, The National Center on Dispute Resolution, is a project funded by the 
United States Department of Education, Office of Special Education Programs. They provide 
technical assistance to state departments of education on implementation of the mediation 
requirements under IDEA '97. CADRE also supports parents, educators and administrators to 
benefit from the full continuum of dispute resolution options. Currently under review, available 
in 2002 from CADRE, P.O. Box 51360, Eugene, OR 97405-0906; or from: 
http : //www . direction service . or g/cadre/ 

Evaluating Alternative Dispute Resolution Programs, by Lee Scharf. This practical 19-page 
document is Chapter 8 of the Federal Alternative Dispute Resolution Program Manager's 
Resource Manual. It is available as a PDF file from the Interagency Alternative Dispute 
Resolution Working Group (IADRWG) home page, as a separate chapter or as part of the entire 
Resource Manual. The Working Group was established to coordinate, promote, and facilitate the 
effective use of dispute resolution processes within Federal agencies as mandated by the 
Administrative Dispute Resolution Act of 1996 and White House Presidential Memorandum. 
Available from: http://www.financenet.gov/financenet/fed/iadrwg/iadrwg.htm 
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Program Evaluation Kit: Victim Offender Mediation Programs, by Mark Umbreit. 
Minneapolis, MN: Minnesota Citizens Council on Crime and Justice, 1992. This guide includes 
brief introductory text explaining the importance of evaluation, and then provides a sample 
questionnaire designed to be administered by program staff without need for professional 
evaluators to assist them. Available for download as a PDF file from the Center for Restorative 
Justice and Peacemaking from: http://ssw.che.umn.edu/rjp/Resources/Resource.htm 

Evaluating Your Conflict Resolution Education Program: A Guide for Educators and 
Evaluators, by Tricia Jones and Dan Kmitta, October 2001. 

This Evaluation Guide, developed by the Ohio Commission on Dispute Resolution and the Ohio 
Department of Education focuses on methods that can be used by school conflict resolution 
program grantees to evaluate program effectiveness and to assess program impact at the building 
level. The authors prepared the manual as a workbook so that it should be easy to use. 
Throughout the beginning parts of the manual they have included some worksheets to help you 
identify the program goals and evaluation goals you want to emphasize. When they provide 
copies of questionnaires and interview questions they have presented them so you can simply 
copy the forms from the book and use them in your school. Available for online reading or 
download at http://www.state.oh.us/cdr/schools/evaluatingcrep.htm 

Indiana Conflict Resolution Institute Searchable Bibliographic Database. This online 
resource is not about how to evaluate mediation services, but instead provides information on 
existing evaluation reports and articles. The database seeks to provide a comprehensive list of 
empirical field studies and program evaluations on conflict resolution. The citation search will 
examine each record in the bibliography database, and select records according to keywords 
entered in one or more of the following fields: author, title, source, publisher, and/or category. If 
article summaries are available, links to the summaries will be displayed in the citation results. 
While it is constantly under construction, the database includes evaluations of conflict resolution 
including juvenile court, international, employment, environmental, and labor relations 
programs. The list of evaluations continues to grow and should not be considered exhaustive. 
Available from: http ://www. spea . indiana.edu/icri/datali st . htm 



II. General Program Evaluation Resources on the Web 

Action Evaluation Project: http://www.aepro.org This site provides information on a 
participatory assessment process known as action-evaluation. The process entails collaboratively 
articulating goals and objectives among the groups involved in a conflict intervention, including 
those funding it, those organizing and convening the intervention, and the participants 
themselves. The action-evaluator collects this information from the groups and summarizes it 
with the help of a computerized database (available over the Web browser) designed to 
systematize the process and organize the data. This goal articulation takes place at the outset of 
an intervention, allowing the action-evaluator to track how goals of various stakeholders evolve 
and use these goals as a basis for both designing the intervention and evaluating it along the way 
and at its conclusion. 
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Bureau of Justice Assistance Evaluation Web Site: http://www.bja.evaluationwebsite.org 
The BJA maintains an extensive Web site designed to provide a variety of resources for 
evaluating primarily criminal justice programs, but much of the information is applicable to other 
contexts as well. The site includes the Electronic Roadmap for Evaluation, which provides 
instructional materials to assist in planning, designing, and conducting evaluations of programs, 
and a section on evaluation resources. It also contains a bibliography of evaluation materials 
organized by specific evaluation topics. 
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Centers for Disease Control Evaluation Working Group Web Site: 

http://www.cdc.gov/eval/resources.htm This site provides an extensive list of categorized online 
resources related to program evaluation. You will find lots of good information here. Resources 
are divided into the following groups: 

* Ethics, Principles, and Standards 

* Organizations, Societies, Foundations, Associations 

* Journals and On-Line Publications 

* Step-by-Step Manuals 

* Logic Model Resources 

* Planning and Performance Improvement Tools 

* Reports and Publications 

* Suggestions 

Collaborative, Participatory, and Empowerment Evaluation Website: 

http://www.stanford.edu/~davidf/empowermentevaluation.html This website, developed by 
David Fetterman, author of the widely-used book Empowerment Evaluation (see below for full 
reference) provides summary links to a very nice collection of user friendly tools supporting 
program evaluation. These include (often free) software programs and online interactive 
evaluation development guides. 



III. Useful Books on Evaluation Research 

Fetterman, D. M., and others (Eds.). (1996). Empowerment Evaluation: Knowledge and Tools 
for Self-Assessment and Accountability. Thousand Oaks, California: Sage. 

Patton, M. Q. (1990). Qualitative Evaluation and Research Methods. Thousand Oaks, 
California: Sage. 

Patton, M. Q. (1996). Utilization-Focused Evaluation. Thousand Oaks, California: Sage. 

Rossi, P., and Freeman, H. (1993). Evaluation: A Systematic Approach. Thousand Oaks, 
California: Sage. 
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Evaluation Module I 



Mediation Needs Assessment: Evaluating Conflict on Campus 

This module is designed for individuals or teams who are considering developing mediation 
services for an institution of higher education. It is intended to assist you in assessing the need 
for a mediation service on your campus and identifying the most appropriate types of service for 
the campus community. Specifically, you will be able to: 

• gather information concerning the current level of conflict resolution practices; 

• discover the state of general knowledge regarding conflict resolution; 

• identify the types of conflict that occur and the locations in which these conflicts most 
often arise; 

• locate other stakeholders who may act as resource people for a mediation service; and, 

• learn about how members of your campus have been affected by conflict in their 
classrooms, workplaces, recreational, and living spaces. 

The module consists of five research tools. Varying levels of expertise are needed for using each 
tool; for some, little or no knowledge of research methods are necessary, whereas others will 
require skills and expertise in survey research, sampling techniques, or descriptive statistics. It is 
recommended that your team begin by considering which tools will generate the knowledge 
needed to begin forming a mediation service; you may find that only one research tool is needed, 
or you may wish to use all five. 



What is a Needs Assessment? 

In general, a needs assessment is a type of research. It is usually performed before a new 
program is implemented, providing you valuable information about existing conditions that 
could be addressed. For example, conducting a needs assessment during the planning stages for 
a campus mediation program will allow the program sponsors to evaluate the need for conflict 
services on campus. 



How Do You Use the Tools? 

Before you begin collecting your own information on the need for conflict services (using the 
suggested research tools contained in this module), you may wish to consult information sources 
that already exist. These may give you a preliminary idea about the type, intensity, and level of 
conflict on campus, as well as the resources that people in conflict have been utilizing. 
Suggestions for valuable information sources include: 

* Student Judicial Affairs/Honor Board case reports 

* Ombuds annual reports 

* Letters to campus or community newspapers dealing with conflict situations 

* Union grievance records 



Page 1 1 of 68 




* Residence Hall incident reports 

* Campus security statistics 

* Sexual harassment complaints 

* Equal Employment Opportunity (EEO) complaints 

Tread carefully when gathering information from many of the above sources; much of the 
information is sensitive and/or confidential in nature. Gaining access to some types of 
documents may require special permissions and/or agreements on your part regarding the 
specific use of the data. In addition to helping you understand the patterns of conflict on your 
campus, annual reports listing counts or summaries of the kinds of complaints/disputes/cases 
handled by various campus offices can also be helpful for establishing a baseline prior to the 
implementation of your program. Reviewing these figures annually, or after your program has 
been in place for some time, can be revealing, especially if case patterns in other areas change 
after your system goes into service. 

Each research tool comes with instructions and/or examples. In some cases, you may wish to 
adapt or modify the tools to more closely reflect the culture of your campus. Each tool is 
preceded by a description, intended to help your team plan for a needs assessment, that includes 
the following information: (1) the questions that can be addressed; (2) how to use the tool; and, 
(3) insight into what the results may provide. 

Research tools in this needs assessment module include: 

1. “Where would they go?” - case examples to indicate the current state of knowledge 
regarding conflict resolution on campus. 

2. HotSpot Mapping - physically locates the sites of frequent conflict. 

3. Conflict Typology Matrix - determines what types of conflict are being experienced by 
whom, about what, and their level of frequency and intensity. 

4. Stakeholder Analysis Chart - identifies allies in your mission. 

5. Conflict Management Inventory - assesses conflict styles used by individuals on campus 
(e.g., confrontation, emotional expression, conflict avoidance). 
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Needs Assessment Tool #1: “WHERE WOULD THEY GO?” Case Examples 

What Questions May Be Addressed by This Measure? 



This tool allows you to assess the current level of knowledge about conflict resolution that exists 
on your campus. By distributing this measure widely on campus, you will learn what students, 
staff, and faculty believe is the best source for solving interpersonal conflict. 



How Do I Use This Measure? 

The case examples included in this measure are typical of many campus conflicts (although you 
may wish to invent a few more of your own examples that are directly relevant to ongoing 
conflicts on your campus). You may distribute this measure to individuals, or have small groups 
work together as in a brainstorming task. It may be administered in meetings, in classrooms, or 
at other campus events. You may want to include a place on the measure (e.g., checklist) where 
people can indicate their role on campus - student, staff, faculty. There are likely to be few, if 
any, issues of confidentiality around using this tool, although it is wise to indicate that answers 
will be anonymous. 



What Information Will the Results Give Me? 

You will learn about the effectiveness and general awareness of conflict resolution services 
currently offered by your institution. You may learn that there are few places to which campus 
members can turn for aid in resolving disputes. You may be able to discover good sources of 
informal problem- solvers on your campus - people you never knew were even involved in 
conflict resolution. Any of this information should be useful to help you to identify needs for a 
mediation service. 
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“WHERE WOULD THEY GO?” 



This survey presents a number of examples of people who, for one reason or another, are having 
difficulty getting along. After each example, write down where you think they should go to find 
help to solve their problem. If you don’t have any ideas, either guess or write “I don’t know” 
after the example. There are no right or wrong answers and responses will remain anonymous. 



Example 1: 

Jason, a student in Introduction to Psychology, is having a disagreement with his instructor, Dr. 
Reynolds, over his midterm essay exam grade. Dr. Reynolds is not being particularly responsive 
to Jason’s concerns. WHERE could Jason go on campus for assistance with his problem? 

Example 2: 

Jennifer, a second-year student living in the residence hall, is having problems with her 
roommate, Aleisha. They have been arguing over using the room for socializing, playing music, 
and having overnight guests. Jennifer doesn’t feel that she’s getting through to Aleisha. 

WHERE could Jennifer go on campus for assistance with her problem? 

Example 3: 

Marie, who works in the admissions office, recently lost her temper with her co-worker, Roger. 
Marie thinks Roger’s computer screensaver is offensive, and she has overheard him telling 
insensitive jokes to a friend on the phone during his break. WHERE could Marie go on campus 
for assistance with her problem? 

Example 4: 

Kalib, a professor in the School of Business, is involved in a conflict with John, another 
professor, over who should be first author on a paper they both worked on. John refuses to 
discuss the matter further with Kalib. WHERE could Kalib go on campus for assistance with his 
problem? 

Example 5: 

Monica, a departmental secretary, feels she has been given far too much work lately by her new 
boss, Henry, and thinks his expectations of her are unrealistic. She is afraid to confront him 
about her workload. WHERE could Monica go on campus for assistance with her problem? 



Please check whether you are: 

Staff Faculty Student Other 
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Needs Assessment Tool #2: HotSpot Mapping of Conflict on Your Campus 

What Questions May Be Addressed by This Measure? 

This measure allows people to visually identify where conflict occurs on campus. By physically 
locating the “hot spots” where various types of conflict occur, program coordinators and 
planners know where to focus their services. 



How Do I Use This Measure? 

This tool is best administered to several people who are familiar with conflict episodes on your 
campus (e.g., campus security personnel, ombuds representatives, administrators, residence 
monitors). Photocopies of the HotSpot map, labeled with locations appropriate for your campus 
(e.g., library, residences, classrooms), can be given to key informants to identify the intensity of 
conflicts that happen in each place. For areas that see frequent, intense conflict (e.g., the 
campus pub, if there is one), instruct informants to place three (3) large Xs in that spot on the 
map. For areas of low conflict, instruct your informants to place few or no Xs in the 
corresponding map location. This tool may be used in a group setting or by individuals. There 
are likely to be few, if any, issues of confidentiality around using this tool, although it is wise to 
indicate answers will be anonymous. 



What Information Will the Results Give Me? 

If you collect between 10 and 20 “maps” from different key informants (including yourself), 
you should be able to identify the spots where the frequency and intensity of conflict is highest. 
This does not give you information about the type of conflict occurring in such spots, however. 
More digging may be necessary to get this information. 



Page 15 of 68 




Library 



Fraternity /Sorority House 




Residence/Dorms 




Cafeteria 



6 . 

Campus Pub 



Labs 






HOTSPOT MAPPING EXAMPLE 
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Needs Assessment Tool #3: Conflict Typology Matrix 



What Questions May Be Addressed by This Measure? 

This matrix offers you the opportunity to creatively gather information about the types of conflict 
that are being experienced on campus. You may gain an understanding of who is in conflict with 
whom, what the conflicts are about, and the frequency and severity of these conflicts. 



How Do I Use This Measure? 

Like the HotSpot map, which identifies locations of conflict, this matrix is intended to give you 
a quick visual and qualitative representation of conflict on your campus. It may be used either 
with individuals or with groups. It may be performed as an exercise for small groups in a 
classroom or workshop, mailed to a representative sample of campus community members to 
gather anonymous data, or it may be posted on a large wall chart in a heavy-traffic area of 
campus. There are likely to be few, if any, issues of confidentiality around using this tool, 
although it is wise to indicate answers will be anonymous. 

Your respondents will be given a quantity of colored stickers to paste on the chart. Red stickers 
represent very serious conflicts (high intensity); yellow stickers represent moderate-intensity 
conflicts; and green stickers represent low-level, ongoing conflicts. Your participants are 
simply asked to distribute their stickers on the matrix according to the number and level of 
conflicts that they are personally aware of. 

For example: a staff member who deals mainly with faculty and administrators is mostly aware 
of low-level conflicts over working conditions and financial issues. She places green stickers 
next to the “working conditions” and “financial compensation” boxes under “Nature of 
Conflict,” and chooses both “Co-Workers” and “Employees-Bosses” as “Disputants” for both of 
these types of conflicts. 



What Information Will the Results Give Me? 

You should be able to discern by looking at a number of different matrices what peoples’ 
concerns are about conflict on campus. You will be able to get a visual sense of the intensity of 
conflict, the disputants involved, and common reasons that disputes arise. Every campus is 
different; this matrix will help you to identify the key issues on your campus that need to be 
addressed by a conflict mediation service. 
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CONFLICT MATRIX 



What types of conflict are you most aware of that occur on this campus? 

Instructions : Affix your stickers in the boxes that represent the types of conflicts that you are 
most aware of. Going across on the matrix, decide WHO is involved in conflicts. Then, going 
down the matrix, decide WHAT TYPE of conflict is usually experienced. Use: 

• RED stickers for serious conflicts; 

• YELLOW stickers for moderate conflicts; and, 

• GREEN stickers for low-level, or ongoing conflicts. 

All responses are anonymous. Use as many stickers as you like to create a picture of conflict on 
your campus! 



WHO is Involved 



TYPE of Conflict 


Student- 

Student 


Student- 

Faculty 


Faculty- 

Faculty 


Co-workers 


Employee- 

Boss 


Grades 












Working conditions 












Interpersonal problems 












Identity issues (e.g., 
sexism/racism) 












Financial 

compensation 












Intellectual property 












Classroom issues 












Living arrangements 












Personal property 












Other (please name) 
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Needs Assessment Tool #4: Stakeholder Analysis Chart 



What Questions May Be Addressed by This Measure? 

This tool, called the Stakeholder Analysis Chart, is intended mainly for personnel who are 
planning to begin a campus conflict resolution program. It allows you to identify stakeholders, 
allies, and human resources in your community who would be able to provide input, support or 
resources for your program. In addition to helping you gauge the attitudes of those around you, 
it can also help to identify the level of influence these potential resources might have. 



How Do I Use This Measure? 

The Stakeholder Analysis Chart is best completed by two or more people - preferably those who 
are working on a planning committee for a conflict resolution program and/or who are 
knowledgeable about the attitudes and influence of others in the campus community. The group 
working on this task begins by brainstorming the names or positions of those who may have an 
interest, an area of expertise, or a stake in a new conflict resolution program. For example, 
“Dean of Students,” “Swimming Coach,” “Campus Police,” or “GBLT Center” could be listed 
on the chart under Stakeholders. The group then decides whether that person, office, or 
organization would be in favor of, indifferent to, or opposed to the presence of a campus conflict 
resolution program. 

The degree of support or opposition can be indicated on the Chart through the use of plus and 
minus signs (see Chart for example). Additionally, if the group is not entirely certain of its 
analysis, it may document an attitude under “E” for estimate, rather than “C” for confident. 
Along with attitudes, the level of influence a stakeholder has may also be documented on the 
Chart by using H (high levels of power), M (medium), or L (low levels of power over the 
program). Be sure to emphasize that the information will be confidential and anonymous so as 
to get frank, honest assessments from your participants. 



What Information Will the Results Give Me? 

For a group that is planning to develop a new program, systematically recording information 
about projected levels of support or opposition from various stakeholders can be invaluable. 
With this information you may be able to identify different areas of expertise and resources or 
different areas of resistance or apathy. 
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Sample 



Stakeholder Analysis Chart 1 

Use the following symbols for completing the chart below: 



Attitude Analysis 


Influence Analysis 


Certainty of Analysis 


++ strongly in favor 


H high (has veto power) 


E Estimate 


+ weakly in favor 


M medium 


C Confident 


0 indifferent/undecided 


L low 




- weakly opposed 






— strongly opposed 







STAKEHOLDERS 


ATTITUDES 


INFLUENCE 




(E) (C) 


(E) (C) 


Vice-President of Student 
Affairs 


++ 


M 


Dean of Law School 


+ 


M 


Masters of Social Work 
Students 


++ 


L 


Union Representing 
Cafeteria Workers 




M 



1 Based on a hypothetical situation and setting. 
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Needs Assessment Tool #5: Conflict Management Inventory 1 

What Questions May Be Addressed by This Measure? 

The instrument provided here is an edited version of Susan Goldstein’s Conflict Management 
Inventory (see LeBaron and Grundison, 1993). This 75-item standardized measure was designed 
to discover the different styles and feelings about conflict that individuals have when handling 
conflict. It contains five subscales (15 questions in each) exploring, all within the context of 
conflict, the respondents’: (1) feelings and beliefs regarding confrontation; (2) emotional 
expression; (3) public/private behavior (4) conflict avoidance; and, (5) self-disclosure. 



How Do I Use This Measure? 

This tool has been developed as a standardized personality instrument, and therefore it is 
recommended that you not adapt it for your own purposes, but rather distribute it as is. 
Participants are asked to rate the degree to which a statement reflects their approach to conflict 
resolution, using a scale from 1 (strongly disagree) to 7 (strongly agree). To control for 
something known as response bias , items for each subscale (i.e., the themes identified by each of 
the five subscales outlined above) were balanced: for half the questions a rating of ‘1’ reflects a 
“higher” score on the subscale while for the other half a higher score is associated with a rating 
of ‘7’. Thus, when analyzing the questionnaire data, half of the response ratings to each 
subscale must be reversed so that all scores end up going in the same direction (i.e., so that a 
rating of ‘7’ has a parallel meaning for all subscale questions). This is a fairly common practice 
in survey design and analysis. There are likely individuals on campus who can provide help if 
you need it. 

To make sense out of the information you get back, you will want to ask for some basic 
demographic information on the people responding to the questionnaire (e.g., age, department or 
major, ethnicity, role on campus - undergraduate, graduate student, faculty, staff). It is 
important to make sure that the survey is sent to (and received back from) people from all facets 
of the campus: students, staff, faculty, and administrators. You may also ask clubs and 
organizations to distribute it. Be advised that in most college settings, collecting information 
such as this from human participants requires that you submit the survey, your purpose for using 
it, as well as a description of what you plan to do with the data, to an Institutional Review Board 
(IRB) to get ethical clearance. Check your university’s IRB policies before beginning to 
distribute this survey. 

There are several important issues about using surveys: selection procedures, confidentiality of 
responses, response rates, and data analysis. It is strongly recommended that a trained researcher 
who is aware of these issues take responsibility for disseminating and analyzing data from this 
tool. Trained researchers may include faculty or graduate students from social science or 
business departments, or senior undergraduates who have been trained in survey research 
methods. 



'Developed by Susan Goldstein, 1990 
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What Information Will the Results Give Me? 

The results will suggest the variety of styles with which people on your campus approach 
conflict situations. If you discover that a great number of them are, for example, conflict 
avoiders, this information may assist you in planning your mediation service. This scale may 
also be used as a pre-test/post-test measure, used among the campus at large (or among potential 
mediators) both before, and again after, a conflict resolution service has been established. A pre- 
test/post-test allows a researcher trained in hypothesis-testing methods of research to compare 
whether, and how, conflict styles change over time. 
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Conflict Management Inventory 

Please indicate how much you agree with each of the following 
statements. Use the scale located to the right. 

(All answers are anonymous) 

1) I feel more comfortable having an argument over the phone 

than in person. 

2) I would be embarrassed if neighbors heard me argue with 

a family member. 

3) In a dispute, I try not to let the other person know what I am 

thinking. 

4) I wait to see if a dispute will resolve itself before taking action. 

5) It is a waste of time to involve emotions in a dispute. 

6) I have arguments. 

7) There are not many people with whom I feel comfortable 

expressing disagreement. 

8) When arguing with someone I feel more comfortable sitting 

side-by-side than face-to-face. 

9) Getting emotional only makes conflicts worse. 

10) In an argument, I try to reveal as little as possible about 

my point of view. 

11) I feel uncomfortable seeing others argue in public. 

12) If my neighbor were having a party that made too much noise I 

would rather call the police than speak with the neighbor myself. 

13) I feel annoyed when someone I am arguing with pressures me to 

talk about my thoughts or beliefs. 

14) If I become angry it is because I have lost control. 

15) I rarely have arguments with my friends. 

Survey continues on next page .. . 



1 = Strongly Disagree 

2 = Disagree 

3 = Disagree Somewhat 

4 = Neutral 

5 = Agree Somewhat 

6 = Agree 

7 = Strongly Agree 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 
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Conflict Management Inventory Cont. 

Please indicate how much you agree with each of the following 
statements. Use the scale located to the right. 



16) It would not bother me to have an argument in a restaurant. 

17) In a dispute there are many things about myself that I will not 

discuss. 

18) Arguments can be fun. 

19) Showing your feelings in a dispute is a sign of weakness. 

20) I prefer to express points of disagreement with others by writing 

them notes rather than speaking with them directly. 

21) Ido not mind being involved in an argument in a public place. 

22) If a friend owed me money I would hint about it before asking 

directly to be paid. 

23) I avoid arguments. 

24) When I am having a dispute with someone, I do not pay 

attention to whether others are around. 

25) I dislike when others have eye contact with me during an 

argument. 

26) I feel uncomfortable when others argue in my presence. 

27) It makes me uncomfortable when other people express their 

emotions. 

28) I feel uncomfortable when others reveal personal thoughts or 

beliefs during a dispute. 

29) If I were upset with a friend I would discuss it with someone else 

rather with than the friend who upset me. 

30) I do not want anyone besides those involved to know about an 

argument I have had. 

Survey continues on next page .. . 



1 = Strongly Disagree 

2 = Disagree 

3 = Disagree Somewhat 

4 = Neutral 

5 = Agree Somewhat 

6 = Agree 

7 = Strongly Agree 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 
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Conflict Management Inventory Cont. 

Please indicate how much you agree with each of the following 
statements. Use the scale located to the right. 



31) I prefer to guess what someone is upset about rather than ask 
about it. 



1 = Strongly Disagree 

2 = Disagree 

3 = Disagree Somewhat 

4 = Neutral 

5 = Agree Somewhat 

6 = Agree 

7 = Strongly Agree 



1 2 3 4 5 6 7 



32) I rarely state my point of view unless I am asked. 

33) I am drawn to conflict situations. 

34) During a dispute I state my opinions openly. 

35) I avoid arguing in public. 

36) In an argument, I feel comfortable expressing my needs and 

concerns. 



1 2 3 4 5 6 7 
1 2 3 4 5 6 7 
1 2 3 4 5 6 7 
1 2 3 4 5 6 7 

1 2 3 4 5 6 7 



37) I hide my emotions in a dispute. 



1 2 3 4 5 6 7 



38) If I am upset about something a friend has done I wait as 

long as possible before discussing the issue. 

39) In a dispute, I want to know all about the other person's 

thoughts and beliefs. 

40) I do not mind when others start arguments with me. 

41) In a dispute, I am glad when the other person asks me about my 

thoughts or opinions. 

42) I feel like running away when people start showing their emotions 

during an argument. 

43) When I have a conflict with someone I try to resolve it by being 

extra nice to him or her. 



1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 



44) In a conflict situation I feel comfortable expressing my thoughts no 
matter who the others involved are. 



2 3 4 5 6 7 



45) It does not bother me to be in a situation where others are 1 2 3 4 5 6 7 

arguing. 

Survey continues on next page .. . 
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Conflict Management Inventory Cont. 

Please indicate how much you agree with each of the following 
statements. Use the scale located to the right. 



46) I enjoy challenging the opinions of others. 

47) For me, expressing emotions is an important part of settling 

disputes. 

48) I prefer to solve disputes through face-to-face discussion. 

49) I am annoyed when someone refuses to discuss a disagreement 

with me because there are others around. 

50) I avoid people who express their emotions easily. 

51) I would feel uncomfortable arguing with one friend in the 

presence of other friends. 

52) I often start arguments. 

53) If a co-worker were interfering with my performance on the job I 

would rather speak to him or her directly than to tell the boss. 

54) In a dispute, I express my emotions openly. 

55) I find conflicts exciting. 

56) I do not like when people ask me to discuss my emotions 

in a dispute. 

57) I expect a family member to know what is on my mind without 

my telling him or her. 

58) Everything should be out in the open in an argument, including 

emotions. 

59) I am just as comfortable having an argument in a public place 

as in a private place. 

60) It annoys me when I know someone is upset with me but he or 

she will not discuss it. 

Survey continues on next page .. . 



1 = Strongly Disagree 

2 = Disagree 

3 = Disagree Somewhat 

4 = Neutral 

5 = Agree Somewhat 

6 = Agree 

7 = Strongly Agree 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 
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Conflict Management Inventory Cont. 

Please indicate how much you agree with each of the following 
statements. Use the scale located to the right. 

61) When something I have purchased is found to be defective, I 

keep it anyway. 

62) It shows strength to express emotions openly. 

63) I would not mind if a friend told others about an argument that 

we had. 

64) I feel frustrated when others discourage my emotional expression. 

65) When involved in a dispute I often become silent. 

66) Arguments do not bother me. 

67) After a dispute with a neighbor, I would feel uncomfortable seeing 

him or her again even if the conflict had been resolved. 

68) An argument can be resolved more easily when people express 

their emotions. 

69) Conflicts make relationships interesting. 

70) I do not mind strangers arguing in my presence. 

71) I like when other people challenge my opinions. 

72) I feel comfortable when other people express their emotions 

during a dispute. 

73) I avoid conflict. 

74) I argue in public. 

75) I feel upset after an argument. 

Please add any comments and thank you . 



1 = Strongly Disagree 

2 = Disagree 

3 = Disagree Somewhat 

4 = Neutral 

5 = Agree Somewhat 

6 = Agree 

7 = Strongly Agree 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 

1 2 3 4 5 6 7 
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Evaluation Module II 



Process Monitoring: Evaluating Program Implementation 

This module is for use by individuals or a team who wish to monitor an existing program. It is 
intended to provide a framework for assessment. Specifically, it will help you to: 

• review systems already in place for implementation and management (e.g., intake); 

• assess whether program is going according to plan and identify areas to improve; 

• identify possibilities for expansion; 

• review roles and responsibilities of your own program in relationship to other campus 
conflict-related office; and, 

• evaluate the decision-making and internal dynamics of your own conflict-related 
office. 



What is Process Monitoring? 

Process monitoring is a type of research that is used to structure the evaluation of an existing 
program. The emphasis is on tracking the day-to-day efforts of a program’s implementation. It 
does not focus on the impact or outcomes, only on the manner in which the program is being put 
into action. Information gained from such an evaluation provides useful strategic information for 
strengthening or expanding the scope of an existing program. 



How Do You Use the Tools? 



Four research tools are included in this module. Each tool focuses on a different aspect of 
program implementation and includes a series of questions to be asked. Process monitoring is 
usually conducted by staff, volunteers, and/or administrators internal to your program. Data 
collected could be directed to and of use by internal office personnel, other campus conflict- 
related programs, campus decision-makers, the wider academic community, and the community 
of individuals working in conflict resolution. 

Research tools in this process monitoring module include: 



1. Standard Operating Procedure (SOP) Review - examines existing protocol for 
implementing your program. 

2. Design new policies and procedures - guides the development of new rules and 
guidelines for implementation. 

3. Structures of responsibility- examines the roles, responsibilities, and relationships among 
different conflict-related efforts on campus. 

4. Internal decision-making processes - assesses the planning process in your office. 
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Process Monitoring Tool #1: Standard Operating Procedure (SOP) Review 

What Questions May Be Addressed by This Measure? 



This tool assesses the efficiency and effectiveness of systems that are currently being used by 
your mediation service office. These include any policies that have been developed (e.g., 
confidentiality) and/or procedures created for conducting the work of your program (e.g., Intake 
forms). 



How Do I Use This Measure? 

To start, you will need to select a procedure or policy you wish to evaluate. The questions below 
provide a template for you to follow; modify as needed. It may be useful to gather any 
paperwork (e.g., forms, templates) relevant to the policy or procedure you plan to evaluate. It 
may also be useful to involve several people in the evaluation process. For example, if you are 
assessing the Intake process, you might wish to include individuals currently in charge of intake 
as well as those who have been responsible in the past (if they are still around). Additionally, 
some questions may be best addressed through interviews and/or self-assessment surveys with 
pertinent individuals (e.g., disputants, office staff, volunteers). Be sure to address the issue of 
anonymity and confidentiality. 



What Information Will the Results Give Me? 

You will be able to determine whether current approaches to the implementation of your 
program are still useful, time-efficient, and cost-effective. You will also identify specific areas 
for improvement. 
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Review Standard Operating Procedures 

Policies and procedures vary from mediation program to mediation program. However, a list of 
some that are commonly found in campus conflict resolution centers is provided below. Select 
one to evaluate using the guidelines below. 

• Intake time and process • Confidentiality 

• Response time • Mediation process 

• Mediator assignment • Reporting 

• Definition/type/style of mediation model used to resolve conflicts 

1. Specific Steps: think about the following two areas: 

a. How do you implement your policy or procedure? 

b. What specifically is involved? 

Listed below are sample questions to ask about a specific policies and procedures. These can be 
used as a guide for evaluating your selected policy/procedure; modify as needed. 

Intake Time and Process 

a. What formal steps are involved in processing an Intake call from first contact to follow- 
up? 

b. What informal steps are involved? 

c. What forms, if any, are used? 

d. How many people are involved? What are their roles? 

Confidentiality 

a. Who explains confidentiality policy to disputants? 

b. When is this explanation provided? 

c. How is it provided (e.g., verbally, written)? 

d. How, when, and by whom is confidentiality discussed with office-relevant persons, e.g., 
mediators, volunteers, staff? 

e. What forms, if any, are used when explaining confidentiality to disputants/office persons? 
Response Time 

a. How much time (hours, days) passes before an initial query is responded to by someone 
in your office? 

b. What is the range of response time from intake call to mediation? 

Mediation Process 

a. Describe the style of mediation or model of intervention used by mediators and 
volunteers. 

b. How does an intake call move to mediation? Outline the steps involved. 

c. What steps and policies are involved in conducting mediation in person? 

d. What steps and policies are involved in conducting mediation over the phone? 

e. What steps and policies are involved in conducting mediation via e-mail? 

f. What steps are involved in explaining mediation and rules of mediation to participants? 

g. What is the range of time involved for mediations from beginning to final agreement? 
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Mediator Assignment 

a. What are the standard policies concerning mediation by volunteers on your campus? 

b. How are mediators (paid and volunteer) selected to be mediators with your program? 

c. What criteria and process are used to assign a case to a mediator? 

Reporting 

a. What data about the disputants are collected? Saved? 

b. How are these data saved? 

c. What policies/procedures are in place concerning client written information? 

d. What format is used for reporting client data (e.g., # mediations, ethnicity of clients)? 

2. Credibility of Policy: Review the steps identified in Step 1. Which, if any, contribute to 
your office having trust in the procedure/policy? Another way to think about this is what makes 
this policy/procedure credible? 

3. Goals of Policy: Think about the following two questions. See Table 1 for examples. 

a. What are the main goals of this policy or procedure? 

b. How will you know when you have achieved the specified goal(s) (i.e., what are the 
visible criteria or measures that indicate a goal has been met?). 

Table 1: Examples of policies/procedures, their goals, and achievement criteria 



Policy/Procedure 


Goal/Why have it 


Achievement Criteria 


Confidentiality Policy 


Client privacy 


No leaks of information 


Mediator training 


Mediator integrity 


High client satisfaction 
No mediators in jail 


Intake process 


Speedy service response 
Client empowerment 


Shorter time from intake to mediation 
High client satisfaction 


Presentations 


Get the word out 


More requests for information 



4. Optional Questions: The following questions are intended to provide greater focus for your 
evaluation. Which to use is somewhat dependent on you goals/reasons for conducting a process 
evaluation and for whom the data are gathered. 

If goal is: 

a. Clarification - Consider whether the current way of doing things would provide the necessary 
information for: 

mediators/volunteers who are supposed to implement the policy/procedure 
disputants/clients participating in the process 

To collect this type of information consider using either an interview or self-report survey with 
mediators and clients. 
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b. Client Comprehension - consider whether: 

clients have a comprehensive understanding of the mediation process in general; and, 
clients have a comprehensive understanding of specific questions they might have. 

To collect this information consider asking your clients through interview or self-report 
techniques. 

c. Procedural Integrity - Consider whether: 

you have an internal review process for making sure steps are being followed 
internally by staff and volunteers; 

you have an external review process the procedures by clients and stakeholders; and, 
you know whether steps are being followed. 

To collect this information consider doing an internal review with mediators/department staff or 
an external review with clients/stakeholders. 



5. Final Re-evaluation - Now that you have conducted an evaluation of the steps involved in a 
particular policy or procedure, consider whether any of the steps require revision/change in light 
of this evaluation data. 
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Process Monitoring Tool #2: New Policy and Procedure Development 



What Questions May Be Addressed by This Measure? 

This tool assesses the development of new work or issues in existing areas of work. 



How Do 1 Use This Measure? 

Based on the above evaluation process or ongoing campus assessments of conflict, use the 
following questions to evaluate how you implement new procedures and policies. You might 
want to consider having several individuals within your program answer the questions and then 
compare responses. Be sure to address the issue of anonymity and confidentiality. 



What Information Will the Results Give Me? 

Results will provide insight into how new policies are currently developed within your program. 
The information can be useful for clarifying how this is done and identifying procedures to 
change. 
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New Policy and Procedure Development 



1. Please list and describe any new or innovative procedures or policies that you have 
considered for implementation in the past year. 



2. Describe which procedures or policies were selected to implement. 



3. Describe your rationale for selecting those procedures or policies to implement first. 



4. What is the status of implementation for the selected procedures/policies? 



5. Describe your plan or time-line to implement the other procedures or policies at a later time. 



NOTE : Use Tool #1 to evaluate the policies and procedures associated with newly adopted 
programs or policies and to re-evaluate those programs or policies for future adaptation. 
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Process Monitoring Tool #3: Structures of Responsibility 

What Questions May Be Addressed by This Measure? 

This tool assesses the roles, responsibilities, and relationships of individuals within your 
program. It also explores the relationship among different conflict-oriented groups on your 
campus. 



How Do l Use This Measure? 

Select the roles, responsibilities, relationships and/or structures deemed important for your 
evaluation and for whom the data is gathered. Use the questions outlined in the tool to guide 
your evaluation. Some questions may be best addressed through interviews and/or self- 
assessment surveys with pertinent individuals (e.g., disputants, office personnel). Be sure to 
address the issue of anonymity and confidentiality. 



What Information Will the Results Give Me? 

The information collected with this tool will help you assess whether the current structure of 
responsibilities within your program is effective, efficient, and meeting the needs of both the 
individuals and the program. It can be used to identify areas of overlap among staff members, 
volunteers, and other individuals involved with the program. Additionally, the insight gained 
into your own program’s relationship with other conflict-related programs and groups on campus 
will suggest ways to reduce overlap, clarify opportunities for partnership opportunities, and 
identify relationships to improve. 
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Roles, Responsibilities, and Relationships 



Questions to ask about People : 

a. What is the nature of this person’s/ these persons’ role(s) (e.g., mediator, volunteer, manager, 
administrator, student assistant)? 



b. Is this their only role and if not, what else do they do? You might want to include roles and 
responsibilities both internal and external to your office. For example, if you have a student 
who volunteers as a mediator, they have numerous responsibilities outside of their role as 
mediator. 



c. Do they feel supported in this role? 



d. Where does the support come from? 



e. What are the kinds of support they need? 



f. Are there any barriers to the carrying out of this role effectively within the institution? 



Questions to ask about Conflict-related Efforts 

a. Is there overlap with other conflict-related offices on campus? If so, what kind? 

b. How are relationships with other conflict-related groups and offices within the institution? 
How have they been beneficial? How have they not been beneficial? 



c. What does this evaluation data tell you about the types of changes, enhancements, and 
planning that are needed? 
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Process Monitoring Tool #4: Internal Decision-Making Processes 

What Questions May Be Addressed by This Measure? 

This tool is designed to assist you in evaluating the internal dynamics of your own program. 



How Do I Use This Measure? 

Below are suggested questions to use. You may wish to consider using interviews, self-report, 
and/or individual self-reflection techniques to collect the information. Be sure to address the 
issue of anonymity and confidentiality. 



What Information Will the Results Give Me? 

Information collected with help you assess the planning process currently being utilized in your 
program. Insight related to priority setting efforts and the decision-making process pertaining to 
program implementation and development will suggest areas for refinement, modification, and 
change. 
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Internal Decision-Making Processes 



a. What was the last procedure or policy that was implemented in your program? 



b. How were the decisions to implement this procedure or policy handled? 



c. How much consultation did the decision-maker do? 



d. With whom did the decision-maker consult? 



e. How do the other individuals working in the program feel about the process of decision- 
making? 
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Evaluation Module III 



Summative Evaluation: Assessing Outcomes and Impact 

This module is designed for individuals or teams who want to review an existing mediation 
program. It is intended to help collect information useful for determining the program’s future 
directions. Specifically, you will be able to: 

• collect information on the effectiveness of mediation for different types of conflict; 

• gain insight into the performance of mediators during a mediation session; 

• gauge satisfaction with and durability of mediated agreements; 

• determine who is contacting and utilizes the services of your mediation program; and, 

• identify areas for improvement. 

The module focuses on three aspects of mediation services: the disagreements, the disputants, 
and the mediation itself. A questionnaire-based survey is the suggested format to be used to 
collect relevant information, although other approaches (e.g., focus groups, interviews) could be 
used. Surveys are particularly useful for summative evaluation because they take little time to 
complete, do not require extensive training to administer, allow for the collection of information 
on a wide range of topics at once, and provide findings that may be summarized and presented 
easily and clearly. Included below is a discussion of how and to whom to distribute surveys and 
some guidance on the analysis of the information collected. 



What is Summative Evaluation ? 

Summative evaluation is one of two broad categories of research. [The other category is called 
process or implementation research. See module II for details on how to conduct this type of 
evaluation.] Summative research examines the outcomes of the program to see if they meet 
expectations. It focuses on determining whether the program is meeting its stated objectives, 
whether it is worth continuing or expanding, and how effective it is. 



How Do You Use the Tools? 

You will find three research tools in this module. Within each tool there are a number of 
suggested survey questions, which reflect current standards and practices in the evaluation of 
mediation. They are intended to serve as a model or template for program managers; you may 
wish to adapt or modify them to more closely reflect the features specific to your program 
and/or the culture of your campus. Although the questions are arranged by topic, please note 
that when you create a survey you may wish to intersperse the questions from the different topic 
areas rather than leaving them clustered by topic. Each tool is preceded by a description, 
intended to help your team plan for the evaluation, that includes the following information: (1) 
the questions that can be addressed; (2) how to use the tool; and, (3) insight into what the results 
may provide. 
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Be advised that in most college settings, collecting information from human participants requires 
that you submit the survey, your purpose for using it, as well as a description of what you plan to 
do with the data, to an Institutional Review Board (IRB) to get ethical clearance. Check your 
university’s IRB policies before beginning to distribute this survey. 

Research tools in this summative evaluation module include: 

1. Case Characteristics - information about the type of cases and how they are managed. 

2. Participant Characteristics - information about who utilizes your mediation services. 

3. Does Mediation Work? - information about the outcomes and process of mediation 

sessions and the performance of your mediators. 

A note about survey distribution : It is recommended that, whenever possible, program managers 
survey the participants in all mediated cases. This relates to the issue of sample size. If an 
evaluation project has only a handful of completed questionnaires to analyze, the themes or 
trends in the data reflect only a few participants’ attitudes, and these may not be representative of 
the typical participant. As a general guideline, evaluators should strive to get as many surveys 
completed as possible. This provides a fuller picture of the effectiveness of your program. 

Questionnaires may be distributed in person or by mail. Each method has its benefits and 
drawbacks. While it is most cost-effective to ask the participants to complete the questionnaire 
before leaving the mediation session, this may lead to responses which are influenced by the 
short-term emotional state of the participant, for example, feelings of accomplishment or feelings 
of exhaustion. Another potential drawback stems from the choice of who distributes the 
questions - if the mediator hands them out at the conclusion of a session, participants may be 
concerned that the mediator will see their responses and be reluctant to comment candidly about 
the mediator’s services. This may be addressed by having the mediator provide an envelope 
along with the survey, with instructions on how to return it to the mediation program manager. 

Questionnaires distributed by mail avoid the complications of those distributed in person, 
but are more costly due to mailing expenses. When mailing questionnaires, it is strongly advised 
that programs include a pre-stamped, pre-addressed envelope in which to return the completed 
questionnaire. A potential drawback of mailed questionnaires - especially follow-up 
questionnaires administered a few months after the mediation - is a low response rate. A cover 
letter emphasizing the importance of the evaluation process will help to increase the response 
rate, but even with this letter, mailed questionnaires seldom receive the response rate that those 
distributed in-person receive. Often, a reminder phone call or postcard will encourage responses 
from individuals who have not returned their surveys. 

An alternative approach is the use of online surveying. There are many advantages to online 
questionnaires, including the savings related to mailing costs and the time required to process 
and compile the responses. Once a mediation provider or education agency has a page on the 
World Wide Web, the development of an online questionnaire is not difficult. A potential 
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downside of relying on online resources is that some individuals may not have easy access to the 
Internet. 

A note about survey data analysis : Most of the information collected by mediation surveys is 
attitudinal - that is, the survey instruments are designed to gauge the perspective or feelings of 
the respondent. A common form of attitudinal survey is the ordinal rating scale, which presents 
responses in some order like bad-to-good or dissatisfied-to-satisfied. Social scientists usually 
order the responses from negative to positive as a means to ensure that a respondent is reading 
through the various degrees of opinion, instead of simply checking the first box. 

To facilitate accurate data gathering, surveys should have code numbers next to each response. 
Without these numbers, recording survey results is tedious and prone to error. In the sample 
questions presented in this module, note that every response has such a code: 

[ ]1 Very dissatisfied 
[ ]2 Somewhat dissatisfied 
[ ]3 Somewhat satisfied 
[ ]4 Very satisfied 

The numbers themselves are only placeholders and do not have a numeric value in relation to 
each other. Thus, a ‘4’ is not twice as good as a ‘2’, as demonstrated in the examples provided 
below. 

There are numerous software programs available in which to record the data collected, although 
any database or spreadsheet will serve the same purpose. There are also many ways to analyze 
the data from rating scales, some of which are complex and require a high degree of 
sophistication with statistical analysis. As this module is primarily concerned with identifying 
opportunities for improvement, extensive statistical analysis will typically not be justified. A 
simple analysis of tabulated responses will reveal those program areas that require attention. 
Consider the following two response sets, based on a survey of thirty-four mediation 
participants. 

Example : 





Question 1 


Question 2 


Very Dissatisfied [ ] 1 


1 


4 


Somewhat Dissatisfied [ ]2 


3 


15 


Somewhat Satisfied [ ]3 


9 


10 


Very Satisfied [ ]4 


21 


5 



For Question 1, the responses are mostly positive, with 30 respondents answering either 
‘Somewhat satisfied’ or ‘Very satisfied,’ and only 4 answering ‘Somewhat dissatisfied’ or ‘Very 
dissatisfied.’ Contrast that distribution with the responses to Question 2, which yielded 15 
responses of ‘Somewhat satisfied’ or ‘Very satisfied,’ but 19 responses of ‘Somewhat 
dissatisfied’ or ‘Very dissatisfied.’ The high proportion of negative responses demonstrates that 
the program area addressed in Question 2 needs attention and improvement. 
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Summative Evaluation Tool #1: Case Characteristics 



What Questions May Be Addressed by This Measure? 

These questions allow you to collect information specific to the conflicts and disputes that your 
office is handling. Details include the types of disputes that come to your office and how they 
are managed, including routing (e.g., whether referred out to another office), time involved, and 
resolution. 



How Do I Use This Measure? 

Case characteristics should be recorded as early and often as possible. Incorporating questions 
into the various systems you have in place or are creating is an excellent way to ensure this type 
of data is collected. For example, during Intake, include questions about how the disputant heard 
about your office, the type of conflict that is occurring, and what other avenues they have tried 
before contacting you. This tool contains examples of characteristics you might want to track. 
You will want to think through issues of confidentiality. 



What Information Will the Results Give Me? 

Results provide important information for both report writing as well as for identifying areas for 
improvement. You can use the information to find out what marketing venues best get the word 
out about your services, the type of conflicts occurring on campus, as well as insight into 
whether cases are being managed efficiently. 
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Case Characteristics - Sample Questions 



Type of Dispute - A variety of different kinds of conflict can occur on a campus, thus this 
category is quite broad. A checklist format is helpful for data collection during Intake and 
possibly during a mediation session. The list provided below is a template; modify as needed. 

housing noise 

harassment/assault vandalism 

borrowed/stolen property course-related (grades, evaluation) 

interpersonal (personal differences not captured by other types) 

Other: 



Referrals - Basically you want to identify how people are finding out about your services as well 
as to whom you refer cases, for either additional or different services. Again, a checklist format 
will likely be the most useful approach. Also, documenting this information during Intake is 
probably most practical. The list provided below is a template; modify as needed: 

residence hall staff campus police/security 

judicial affairs campus clergy 

campus legal services faculty member 

media (newspaper, phonebook, public service announcement) 

roommate/friend/acquaintance (aka word-of-mouth) 

Other: 



Other Avenues Attempted - What else has a disputant tried? This information can help gauge 
how well the word about mediation is reaching your campus audience and how mediation 
compares with other possible conflict-resolution options (e.g., talking about it with a friend, 
arbitration, legal action). This is an open-ended question to be asked during Intake. 



Pending Action - Sometimes cases come to mediation with strings attached. For example, there 
may be disciplinary actions or a small claims court filing that may be or already have been 
initiated. This is an open-ended question to be asked during Intake. 



Time Involved - There are several issues of interest pertaining to mediation sessions as well as to 
the impact of the conflict on the disputants’ and your time. 

• Mediation Sessions: Document the start and end time for each mediation session to 
determine the number of minutes/hours spent in mediation. Also, document the 
number of sessions devoted to each conflict. For example, a certain conflict might 
involve two mediation sessions to reach resolution, each lasting an hour. 

• Days Devoted to Conflict Resolution: Document the date you first received a case 
and the date the case was closed to determine the number of days your office was 
involved in the conflict. If possible, try and find out the date the conflict started 
between the disputants to measure the number of days between inception of conflict 
and case closing. 
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Summative Evaluation Tool #2: Participant Characteristics 



What Questions May Be Addressed by This Measure? 

This tool provides insight into who from within the campus community is utilizing your services. 



How Do 1 Use This Measure? 

Participant characteristics should be recorded as early and often as possible. This type of 
information is often collected in a questionnaire/survey format, completed by the participant. 
Incorporating questions into the various systems you have in place or are creating is an excellent 
way to ensure you are collecting this type of data. For example, the questions included in this 
tool could be included in an exit evaluation survey given at the end of a mediation session. This 
tool contains examples of characteristics you might want to track. You will want to think 
through issues of confidentiality. 



What Information Will the Results Give Me? 

Results from this tool also provide important information for both report writing and for 
identification of areas for improvement. For example, it can help identify populations on campus 
who may need to be targeted for outreach about your services. 
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Participant Characteristics - Sample Questions 

Demographic data - These questions provide insight into the types of people using your services. 

The list provided below is a template; modify as needed. 

Age - have disputant write in their actual age or check an age category (e.g., 18-22). 

Gender - useful to have disputant check a box rather than write it in. 

Ethnicity - consult the campus human relations office for appropriate categories. 

Education - you can focus on completed education (e.g., high school diploma, bachelors, 
masters, doctorate), the current year in school (e.g., sophomore, junior, masters, 
doctorate), or both. 

Affiliation/role - provide a checklist of different roles on campus: e.g., student, faculty, 
staff. 

Role in mediation - provide a checklist for the participant. Listed below are different 
roles with an explanation for each. 

Complainant (first party or initiator of contact with the mediation resource) 

Respondent (second party) 

Observer (a mediation center staff or trainee) 

Support person (ranges from advocate for to friend of a disputant) 



Relationship - You are interested in gaining insight into the type and length of the relationship 
among the parties involved in the dispute. A checklist is probably the easiest way to collect this 
information. The list provided below is a template; modify as needed: 

acquaintances landlord/tenant 

classmates employer/employee 

roommates/housemates . co-workers (staff or faculty) 

neighbors . faculty-staff 

friends student-faculty 



dating family (immediate or extended) 

domestic partners/married strangers 

divorced/separated Other: 

ex-boyfriend/girlfriend 



Repeat Player - This provides information about whether a participant has previously utilized 
mediation. A simple checkbox, with a “yes” or “no” answer, will likely suffice. If you are 
interested in learning whether their experience with mediation was with your program, you may 
want to include a question. 

Have you previously participated in mediation? Yes No 

If yes, was it with our office? Yes No 
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Summative Evaluation Tool #3: Does Mediation Work? 



What Questions May Be Addressed by This Measure? 

This tool specifically assesses the mediation component of your program. It includes three 
indicators of performance. See Table 1 below for a listing of questions that can be addressed for 
each area. 



How Do I Use This Measure? 

Each question is preceded by the word(s) ‘Pre’, ‘Exit’, and/or ‘Follow-up.’ These represent the 
most typical time points for collecting information: pre-mediation, immediately following 
mediation (exit survey), and three- or six-month follow-up. All but two of the questions are 
designed as a checklist for which the participant selects the most appropriate answer. The final 
two questions are open-ended, soliciting comments for improvement. The questions are geared 
toward the participants, although some questions might be appropriate for mediators or 
observers to answer as well. You will want to think through issues of confidentiality. 



What Information Will the Results Give Me? 

You will learn about participants’ satisfaction with the services being offered by your program as 
well as the effectiveness of mediation for different kinds of conflicts. Additionally, you will 
obtain feedback on your mediators’ skills and abilities in a mediation setting. This information 
can be useful for modifying the program as well as for report writing to campus administrators or 
other conflict-related offices on campus (e.g., campus security, ombuds). 



Table 1: Performance Indicators for Summative Evaluation of Mediation Session 



Performance Indicators 


Measures 


Effectiveness of Program 


Outcomes of mediation 
Satisfaction with mediated outcomes 
Durability of mediated outcomes 
Impact on relationship between participants 


Mediation Process 


Appropriateness/Usefulness 
Preparation process and materials 
Fairness (opportunity to tell story, feeling 
understood, respectful treatment, control 
over outcome) 


Mediator Performance 


Skills 

Knowledge 

Impartiality 
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Does Mediation Work? 



I. Effectiveness of Program 

A. Mediated Outcomes 

Exit. What was the outcome of the mediation? 

[ ] 1 No agreements 

[ ]2 Agreement on some but not all issues 
[ ]3 Agreement on all issues 

Follow-up. If an agreement was not reached in mediation, how was the matter resolved? 

[ ] 1 Through a hearing [or other means; change as applicable] 

[ ]2 Through an informal agreement or settlement 
[ ]3 The concern was dropped 
[ ]4 The concern remains outstanding 

B. Satisfaction with Outcomes 

Exit , How satisfied are you with the outcome of the mediation? 

Follow-up. 

[ ]1 Very dissatisfied 
[ ]2 Somewhat dissatisfied 
[ ]3 Somewhat satisfied 
[ ]4 Very satisfied 

[If an agreement was reached, the following question may be used] 

Exit, How satisfied are you with the other participant’s fulfillment of any agreements 
Follow-up. reached in mediation? 

[ ]1 Very dissatisfied 
[ ]2 Somewhat dissatisfied 
[ ]3 Somewhat satisfied 
[ ]4 Very satisfied 

C. Durability of Mediated Outcomes 

Follow-up. If an agreement was reached in mediation, has it effectively addressed the issues? 

[ ]1 No 
[ ]2 Yes 

Follow-up. If an agreement was reached in mediation, have the provisions of the agreement 
been followed or implemented? 

[ ] 1 None of the provisions have been followed or implemented 
[ ]2 Some (but not all) provisions have been followed or implemented 
[ ]3 All provisions have been followed or implemented 
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D. Impact on Relationship between Participants 

Exit , When you compare your situation before and after the mediation, how has 

Follow-up. mediation affected your relationship with the other participant(s)? 

[ ] 1 Mediation has harmed the relationship 
[ ]2 Mediation has had little or no effect on the relationship 
[ ]3 Mediation has improved the relationship 

Pre, How would you characterize the relationship among participants? 

Exit, 

Follow-up. 

[ ]1 Very cooperative 
[ ]2 Somewhat cooperative 
[ ]3 Somewhat adversarial 
[ ]4 Very adversarial 

Exit. The other person listened to my views. 

[ ] 1 Strongly agree 
[ ]2 Agree 
[ ]3 Disagree 
[ ]4 Strongly disagree 

Exit. The other person learned something new about my point of view. 

[ ] 1 Strongly agree 
[ ]2 Agree 
[ ]3 Disagree 
[ ]4 Strongly disagree 

Exit. I learned something new about the other person’s point of view. 

[ ] 1 Strongly agree 
[ ]2 Agree 
[ ]3 Disagree 
[ ]4 Strongly disagree 
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II. Mediation Process 



A. Appropriateness/Usefulness 

Exit, How productive or unproductive was mediation for this issue? 

Follow-up. 

[ ] 1 Mediation was very unproductive 
[ ]2 Mediation was somewhat unproductive 
[ ]3 Mediation was somewhat productive 
[ ]4 Mediation was very productive 

B. Preparation Process and Materials 

[Could add specific questions re intake person, brochure, other printed material] 

Exit. Based on the information or materials provided by the mediator or mediation 
agency, how prepared or unprepared did you feel for the mediation? 

[ ]1 Very unprepared 
[ ]2 Somewhat unprepared 
[ ]3 Somewhat prepared 
[ ]4 Very prepared 

C. Fairness of Mediation Process 

Exit. How satisfied were you with your opportunity to relate your issues and concerns 
during the mediation? 

[ ]1 Very dissatisfied 
[ ]2 Somewhat dissatisfied 
[ ]3 Somewhat satisfied 
[ ]4 Very satisfied 

Exit. How satisfied were you with how well you represented yourself in the mediation? 

[ ]1 Very dissatisfied 
[ ]2 Somewhat dissatisfied 
[ ]3 Somewhat satisfied 
[ ]4 Very satisfied 

Exit. How satisfied were you with your level of control over the process during 
mediation? 

[ ]1 Very dissatisfied 
[ ]2 Somewhat dissatisfied 
[ ]3 Somewhat satisfied 
[ ]4 Very satisfied 
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Exit. How satisfied were you with your level of control over outcomes in mediation? 

[ ]1 Very dissatisfied 
[ ]2 Somewhat dissatisfied 
[ ]3 Somewhat satisfied 
[ ]4 Very satisfied 

Exit. Compared to other likely means of resolving this matter, how did mediation 
affect the time spent addressing this matter? 

[ ] 1 Mediation increased the time spent 
[ ]2 No change in time spent 
[ ]3 Mediation shortened the time spent 
[ ]4 Don’t know 



III. Mediator Performance 

A. Skills of the Mediator 

Exit. How well did the mediator understand your issues and concerns? 

[ ] 1 Understood not at all 
[ ] 2 Understood partially 
[ ]3 Understood fully 

Exit. How respectful or disrespectful was the mediator toward you? 

[ ]1 Very disrespectful 
[ ]2 Somewhat disrespectful 
[ ]3 Somewhat respectful 
[ ]4 Very respectful 

Exit. Did you feel pressured by the mediator to reach an agreement? 

[ ]1 Yes 
[ ]2 No 

Exit. Focusing on the skills of the mediator, how satisfied were you with the 
performance of the mediator? 

[ ]1 Very dissatisfied 
[ ]2 Somewhat dissatisfied 
[ ]3 Somewhat satisfied 
[ ]4 Very satisfied 
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Exit. 



Did the mediator offer specific suggestions for resolution? 



[ ]1 No 
[ ]2 Yes 

B. Knowledge of the Mediator [as applicable] 

Exit. How satisfied were you with the mediator’s knowledge of the topic(s) discussed? 

[ ]1 Very dissatisfied 
[ ]2 Somewhat dissatisfied 
[ ]3 Somewhat satisfied 
[ ]4 Very satisfied 

C. Impartiality of the Mediator 

Exit. Concerning the impartiality of the mediator, how did you feel? 

[ ] 1 The mediator favored my party 

[ ]2 The mediator favored the other party 

[ ]3 The mediator was neutral, and favored neither party 

Exit. In case this evaluation has failed to cover them, what actions of the mediator — 
good or bad — were most important to you? 

IV. Overall Assessment 

Exit , Please provide any suggestions for improvement related to the mediation services 

Follow-up. you received. What could be done differently? 
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Evaluation Module IV 



Evaluation of Mediation Training 

This module is designed for individuals or teams who are considering developing mediation 
services for an institution of higher education. It is intended to assist you in evaluating any 
training workshops you offer. Specifically, you will be able to: 

• assess whether training is meeting expectations; 

• determine how effective training is for professional development; and, 

• identify areas for improvement. 



Why Evaluate Training? 

Properly conducted, evaluations allow you to prepare for and conduct useful and appropriate 
training events. Evaluation results also help determine the value and effectiveness of your 
training by comparing training results to your expectations. 



How Do You Use the Tools? 

Each research tool comes with instructions and/or examples. In some cases, you may wish to 
adapt or modify the tools to more closely reflect the culture of your campus. Each tool is 
preceded by a description, intended to help your team plan for the evaluation, that includes the 
following information: (1) the questions that can be addressed; (2) how to use the tool; and, (3) 
insight into what the results may provide. 

Be advised that in many college settings, collecting information from human participants 
requires that you submit the survey, your purpose for using it, as well as a description of what 
you plan to do with the data, to an Institutional Review Board (IRB) to get ethical clearance. 
Check your university’s IRB policies before beginning to distribute this survey. 

Research tools in this training evaluation module include: 

1. Mediation Selection Checklist - a way to gauge how representative your mediators are to 
groups you are serving 

2. Conflict Orientation Survey - assesses how people approach conflict 

3. Mediation Training Evaluation Survey - information about effectiveness of training. 

4. Mid-Training Evaluation Feedback - glean ideas about how to modify training while still 
conducting it. 
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Training Evaluation Tool #1: Mediator Selection Checklist 



What Questions May be Addressed by this Measure? 

The Mediator Selection Checklist is for use in assessing whether mediators are representative of 
the campus population. It identifies pertinent demographics to think about when selecting 
mediators to participate in training. Having mediators that are representative of the population 
they are serving is a crucial factor in successful mediation programs. 



How Do I Use This Measure? 

The list of demographic information presented identifies the standard categories to consider; it is 
not exhaustive and does not include all categories or subcategories. This is deliberate so that you 
can adapt it to the circumstances of your particular campus. If available, select demographic 
categories utilized by other administrative offices on campus. This ensures uniformity. 

First, collect information on demographics (e.g., number students, faculty, staff; percent 
male/female) for the unit to which you are interested in comparing your mediators. For example, 
if you wish to have a cadre of mediators that is representative of the entire campus, get 
demographic information for the whole campus. This could be obtained from campus 
administrative offices, such as, Admissions, Registrar, President or Provosts office. If, however, 
the mediation program will be used solely in residence halls, then the group of mediators would 
need to be representative of the residence hall population, not the entire campus. 

Second, for each category fill in the number of mediators who represent that category. Calculate 
the percent that category represents. For example, if a total of 20 students are in the mediation 
cohort and 15 are Caucasian than the category "Caucasian" will represent 75% of the mediation 
cohort. Ideally the percentages should reflect the percentages of the population of interest. 

A note about reliability and validity : The face validity of the instrument is high. That is the 
items on the checklist look like they are recording the data sought - demographics for a 
mediation training program. The items are pretty universal and the instrument is open so that if 
items need to be added or deleted each program will have the flexibility to do so. There has been 
no reliability analysis for the checklist. 



What Information Will the Results Give Me? 

The findings will help you determine how representative your team of mediators is compared to 
the population you wish to serve. You can use the information to identify whether you need a 
targeted recruitment effort to diversify your group of mediators. 
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Mediator Selection Checklist 



Total Campus Population_ (or other unit of interest, e.g., residence halls, faculty) 

Number of Mediators 

Mediator to Campus Population Ratio : 

For Mediators, List Number/Percentage of: 



Gender 


Gender Orientation 


Females 


Gav 


Males 


Lesbian 




Heterosexual 


Age 


Status 


18-22 


Undergraduate 


23-30 


o 

Graduate 


31+ 


Faculty 




Staff 


Race 


Ethnicity 


African Americans 


Italian 


Native Americans 


Chinese 


Caucasians 


Mexican 


Asian Pacific 




Discipline 


Affiliations 


Humanities 


Sororitv 


Education 


Fraternity 


Environmental Affairs 




Law 




Communications 




Business 




Other: 





Note: Categories provided here are simply suggestions, you may want to modify or add categories as appropriate 
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Training Evaluation Tool #2: Conflict Orientation Survey 



What Questions May be Addressed by this Measure? 

People view conflict in a variety of ways. The Conflict Orientation Survey can help gauge an 
individual’s orientation towards conflict. Conflict orientation is defined as how one conducts 
themselves in hypothetical conflict scenarios. The theory guiding the Orientation Scale is based 
upon Morton Deutsch’s work of positive and negative conflict. At the extremes, people 
demonstrate a positive orientation or negative orientation, with many shades in between. 
Positively oriented, non-violent methods of conflict resolution might include talking, 
cooperating, caring, and thinking about the relationship. Negatively oriented, more violent 
methods of conflict resolution might include the use of physical force, humiliation, or shaming. 



How Do I Use This Measure? 

Both a pre- and a post-test survey are included. They ask the same questions just in a different 
order. Administer the pre-test survey before the training begins. For the post-test survey you 
can either administer it a couple of days after training ends or, if this is not possible (and often it 
is not), right at the end of training. The instrument takes approximately 5 minutes to fill out. 
You will want to think through issues of confidentiality. 



How to score : To control for something known as response bias, some items have been inverted, 
whereby a rating of ‘1’ reflects a “higher” score rather than a rating of ‘5’. Thus, when 
analyzing the questionnaire data, the response ratings for any inverted items must be reversed so 
that all scores end up going in the same direction (i.e., so that a rating of ‘5’ has a parallel 
meaning for all subscales). High scores should indicate a positive orientation; low scores a 
negative orientation. One would hope that after a mediation training those individuals with a low 
pre-test score would move higher on the scale, indicating a shift in the way they approach 
conflict, from negative to positive. 

To obtain an overall score for each individual, follow the steps below. 

a. Reversing Response Ratings (see chart for how to reverse the ratings) 

1. Pre-Test: Reverse the scores for items 2, 3, 5, 6, 9 and 12. 

2. Post-training Survey: Reverse the scores for items 4, 6, 8, 9, 11, and 12. 

Score Reversal Chart: 1 = 5 

2 = 4 

3 = 3 

4 = 2 

5 = 1 
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b. Determining Conflict Orientation 

1. Add the values for the twelve (12) items together, creating a sum score for the person. 
The highest score possible is 60 and the lowest score possible is 12. Do this for both 
the pre- and post-test surveys. 

2. Compute the average score (i.e., divide by 12) for both the pre- and post-test surveys. 
The highest possible average is 5 and the lowest 1. 

3. Subtract the pre-test average score from the post- test average score. Ideally pre-test 
scores will be lower than post-test scores; any difference between the two scores can 
be tied to the training. 

4. If you are savvy in the use of a spreadsheet enter the data and perform any number of 
analyses available on the spreadsheet program. 

A note about Reliability and Validity : This instrument has had some reliability and validity 
work conducted on it. Face validity was addressed by having the instrument reviewed by 
conflict resolution trainers and had an 80% agreement rate. Chronbach’s alpha, a measure of 
internal validity, was reported as .74 in one study conducted by Kmitta, 1996. More work with 
this instrument, across training situations, would be helpful in determining validity. Pre-test 
sensitization and post-test immediacy may skew results. 



What Information Will the Results Give Me? 

The results provide a picture of how your mediators may approach conflict. This is useful for 
evaluating the effectiveness of training with respect to attitude change about conflict situations. 
It is also useful for the selection of and assignment of mediators to a case. For example, if your 
mediation program uses co-mediators, you might want to ensure the mediators have different 
orientations to conflict, rather than the same. 
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Site Date 

Conflict Orientation — Pre-Training Survey (All responses are confidential) 

The following questions concern attitudes about conflict and ways of handling conflict. Please 
answer each question as honestly as you can. 

Please indicate how much you agree with each of the following statements: 



1 = never true 2 = rarely 3 = sometimes true 4 = often 5 = always true 



1. 1 am careful to avoid attacking a person’s intelligence when I critique their ideas. 

2. When someone is stubborn, I often use insults to soften the stubbornness. 

3. If a person I am trying to influence really deserves it, I attack their character. 

4. When I critique a person’s ideas I try not to damage their self-concept. 

5. When people do things that are mean or cruel, I attack their character in order 
to correct their behavior. 

6. When nothing seems to work in trying to influence someone, I yell and scream in 
order to get some movement from them. 

7. 1 am not threatened by conflict. 

8. When people have conflicts they should try to work with the other person to solve it. 

9. Physical fighting is an effective way to deal with conflict. 

10. When I have a conflict with someone I always discuss it with them as soon as possible. 

11. Overall I think I handle conflicts effectively. 

12. Sometimes physically fighting it out is healthy. 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 



Some useful background information: 

1. Your job title/Year in School 

2. Name of organization you work for 

3. Birthdate (month,day,year) 

4. Gender: Female Male 

5. Ethnicity: 

African-American Appalachian Caucasian 

Native American Inter-racial Hispanic 

Asian 
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Site Date 

Conflict Orientation — Post-Training Survey (All responses are confidential) 

The following questions concern attitudes about conflict and ways of handling conflict. Please 
answer each question as honestly as you can. 

Please indicate how much you agree with each of the following statements: 



1 = never true 2 = rarely 3 = sometimes true 4 = often 5 = always true 



1. I am not threatened by conflict. 

2. When people have conflict they should try to work with the other person to solve it. 

3. Physical fighting is an effective way to deal with conflict. 

4. When I have a conflict with someone I always discuss it with them as soon as possible. 

5. Overall I think 1 handle conflicts effectively. 

6. Sometimes physically fighting it out is healthy. 

7. I am careful to avoid attacking a person’s intelligence when I critique their ideas. 

8. When someone is stubborn, I often use insults to soften the stubbornness. 

9. If a person I am trying to influence really deserves it, I attack their character. 

10. When I critique a person’s ideas I try not to damage their self-concept. 

11. When people do things that are mean or cruel, I attack their character in order 
to correct their behavior. 

12. When nothing seems to work in trying to influence someone, I yell and scream in 
order to get some movement from them. 



1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



Some useful background information: 

1. Your job title/Year in School 

2. Name of organization you work for 

3. Birthdate (month, day, year) 

4. Gender: Female Male 

5. Ethnicity: 

African-American Appalachian Caucasian 

Native American Inter-racial Hispanic 

Asian 
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Training Evaluation Tool #3: Mediation Training Evaluation Survey 



What Questions May be Addressed by this Measure? 

This tool is designed to examine whether training successfully contributed to the professional 
development of your mediators and/or trainees. It will answer several questions. Is training 
meeting participants’ expectations? Are they satisfied with the training they are getting? What 
specific aspects of training do they find helpful? Less helpful? What would they change to 
make training even better? 



How Do I Use This Measure? 

Two versions of the same survey are provided, one for pre- and one for post-training. They ask 
the same questions in the same order. The survey is for the participants to complete although 
you may want to develop a modified version for trainers to complete. Administer the pre-test 
immediately before the training begins, in conjunction with other assessments or paperwork. 
Administer the post-test survey immediately at the end of training. You will want to think 
through issues of confidentiality. 

How to score - The instrument is positively skewed, which means there is no need to reverse any 
scores - a high score means high satisfaction and expectations, a low score indicates low. 

Satisfaction (items 1-6 on survey) 

a. Determining satisfaction for an individual participant: 

1. Add the value for items 1-6 together, creating a sum score for the person. The highest 
score possible is 30 and the lowest 6. Do this for both the pre- and post-test surveys. 

2. Compute the average score (i.e., divide by 6) for both the pre- and post-test results. 

3. Subtract the pre-test average score from the post-test average score. 

Positive difference 4 training was satisfactory 
Negative difference 4 training was unsatisfactory 

b. Determining overall satisfaction: 

1. Compute the overall average score by adding together all the individual average 
scores (those computed in #2 above) and dividing by the number of participants. Do 
this for both the pre- and post-test results. 

2. Follow instructions for #3 above. 

Expectations (items A1-A3): 

a. Determining whether expectations were met for an individual participant: 

1. Add the value for items A1-A3 together, creating a sum score. The highest score 
possible is 15 and the lowest 5. 

2. Compute the average score. Do this for both pre- and post-test surveys. 

3. Subtract the pre-test average score from the post-test average score. 

Positive difference 4 training met expectations 
Negative difference 4 training did not meet expectations 
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b. Determining whether expectations were met overall: 

1. Compute the overall average score by adding together all the individual average 
scores (those computed in #2 above) and dividing by the number of participants. Do 
this for both the pre- and post-test results. 

2. Follow instructions for #3 above. 

Changes/Modifications for Future Training (items B1-B3 on survey): 

1. Review all participants’ answers to identify emergent patterns in the responses. 

2. Do this for each of the three (3) questions. 

A note about Reliability and Validity : The instrument has had no extensive validity and 
reliability work conducted on it. Face validity was addressed through review of the instrument 
by conflict resolution trainers but no inter-rater agreement was established. One reliability 
analysis was conducted using an adapted version of the instrument by Kmitta, 2000. 
Chronbach’s alpha on that analysis was a respectful .83. 



What Information Will the Results Give Me? 

The results can help determine whether expectations for skill and basic knowledge acquisition 
are being met. Use the information to identify specific areas of improvement for training in the 
future. 
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Site. 



Date 



Mediation Training - Pre- Workshop Assessment (All responses are confidential) 
Please indicate how much you agree with each of the following statements: 



1 = strongly disagree 2 = disagree 3 = moderate 4 = agree 5 = strongly agree 



1. I feel that I often use what I learn in a training workshop. 

2. I recommend attendance of training workshops. 

3. The amount of material covered in a training workshop is usually appropriate. 

4. Training workshops, regardless of focus, are usually a productive use of my time. 

5. I find the content of training to be more important than the style of the presenter. 

6. In general training workshops are interesting. 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 



Please indicate your level of expectation for each of the following statement: 

1 = none 2 = a little 3 = neutral 4 = high 5 = very high 
Al. I expect to learn basic knowledge about how to conduct a mediation. 1 2 3 4 5 

A2. I expect to develop mediation skills. 1 2 3 4 5 

A3. My overall expectations for this mediation workshop are: 1 2 3 4 5 



Please answer the following short answer questions. Use the back of this page if you need 
more space. 

Bl. What do you like about training workshops? 



B2. What do you dislike about training workshops? 



B3. What would you do to improve a training workshop? 



Some background information: 



Name 




Affiliation (Organization/School) 


Profession/Major 




Years of Education/ Year in School 


Gender 


Female 


Male 



How many training workshops have you attended over the past 12 months?. 
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Site Date 

Mediation Training - Post- Workshop Assessment (All responses are confidential) 
Please indicate how much you agree with each of the following statement: 



1 = strongly disagree 2 = disagree 3 = moderate 4 = agree 5 = strongly agree 



1. 1 feel that I will use what 1 learned in the mediation training workshop. 

2. I would recommend attendance to this mediation training workshop. 

3. The amount of material covered in the training workshop was appropriate. 

4. The mediation training workshop was a productive use of my time. 

5. I found that content was more important than the style of the mediation trainers. 

6. In general the mediation training workshop was interesting. 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 



Please indicate the level of fulfilment for each of the following statement: 

1 = none 2 = a little 3 = neutral 4 = high 5 = very high 

Al. My expectation for knowledge about conducting a mediation was met. 1 2 3 4 5 

A2. My expectation for skill development in conducting a mediation was met. 1 2 3 4 5 

A3. Overall, my expectations for this workshop were fulfilled. 1 2 3 4 5 



Please answer the following short answer questions. Use the back of this page if you need 
more space. 

Bl. What did you like about the mediation workshop training? 



B2. What did you dislike about the mediation workshop training? 



B3. What would you do to improve the mediation workshop training? 



Some background information: 



Name 




Affiliation (Organization/School) 


Profession/Major 




Years of Education/ Year in School 


Gender 


Female 


Male 



How many training workshops have you attended over the past 12 months?. 
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Training Evaluation Tool #4: Mid-Training Evaluation 



What Questions May be Addressed by this Measure? 

This instrument is used to get a rough measure of the participants’ feelings about the training at 
around the mid-point of the training. 



How Do I Use This Measure? 

This tool is most useful for multi-day training sessions. Administer it near the end of the middle 
point of the training. For example, if training is a total of two (2) days, collect information at the 
end of the first day. You will want to think through issues of confidentiality. 

How to score : These items are qualitative and should be reviewed for emergent patterns in the 
responses from the participants. Look for likes, dislike, and suggestions. 

A note about Reliability and Validity : The instrument has some face validity. 



What Information Will the Results Give Me? 

Use the information to identify things that were frustrating to training attendees. Select those 
you can do something about and do it. For example, if there are complaints about the lack of 
breaks, review the remainder of your training schedule and juggle as needed so as to incorporate 
more or longer breaks. Let participants know what you intend to do. Those areas you are unable 
to do anything about (e.g., temperature, amount of material that needs to be covered), 
acknowledge the frustrations and offer explanations. Lastly, for things identified as positive, 
share this as well; it is always good to know what is going well. 
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Mid-Training Evaluation 



Please add comments below each item. 

Training Methods 

a. Lectures - is the material being covered in a comprehensive and understandable manner? 

b. Demonstrations - do the demonstrations help you understand the material presented in the 
lectures? 

c. Exercises - do the exercises adequately provide you with the opportunity to practice skills that 
you are learning? 

Trainers 

d. Are the trainers knowledgeable about the material? 

e. Do they present the material in a manner suitable to your learning style? 

f. Are there methods or approaches that should be stopped, started, or continued? Increased or 

decreased? 

Materials 

g. Are the training manuals adequate? 

h. Are there subjects or skills that you would like to have more of? Less of? 

Facilities 

i. Are the training facilities adequate? 

j. Temperature? 

k. Seating? 

l. Break out rooms / spaces? 

m. Comfortable? 

Please make a list of things to: 

Stop Start Continue 
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Training Evaluation Tool #5: Mediation Role Play Checklist 



What Questions May be Addressed by this Measure? 

The Mediation Role Play Checklist is used to determine the basic demonstrated skill level of the 
participants’ use of mediation. It can be used to assess the mediation ability of trainees. 



How Do I Use This Measure? 

The checklist is to be used during the final role play for a mediation training workshop. For each 
stage of a mediation several important mediator skills have been identified. The evaluator, using 
a scale ranging from 1 (not covered) to 3 (covered), indicates their assessment of how well a 
trainee exhibited a particular skill. Two different examples are included. Use one or the other or 
mix and match to create your own checklist. 

How to score : The instrument is positively scored; a high score indicates greater demonstrated 
skill level. 

1. Add together the scores for the skills listed under each phase of mediation. This 
provides an overall assessment of how well a trainee does in a particular area of 
mediation facilitation. 

2. Add together the scores for each phase of mediation (calculated in #1) to obtain an 
overall score per individual. The highest score possible is 72 and the lowest score 
possible is 24. Note that lower scores could be a result of low performance or due to 
a failure to record an observation. 

A note about Reliability and Validity : This instrument has high face validity but has had no 
formal validity or reliability work up conducted with it to date. 



What Information Will the Results Give Me? 

The results provide information on how adept newly trained mediators are at facilitating a 
mediation process. Areas of strengths and weaknesses can be identified, which can be used as 
the basis for future on-going trainings. It is also useful to share the results with mediators so 
they get insight into their own skill base. Additionally, the information can be used in selecting 
mediators for cases. 
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Sample 1 



Checklist for Mediation Role Plays 

Trainee’s Name: Evaluator’s Name: 

Phase 1: Introduction 



Names of everyone present 

Ground rules 

Confidentiality 

Explanation of mediation 

Phase 2: Telling the Story 



1 = not demonstrated 

2 = alluded to but not clearly demonstrated 

3 = clearly demonstrated 



Listen 

Reflect and paraphrase after each disputant speaks 

Thank and affirm 

Keep ground rules 

Identify issues 

Phase 3: Understanding the problem 

_ Listen for issues 

Listen for possible agreements 

Listen for needs 

Phase 4: Alternative Search 



Point out areas of agreement 

List each issue to address 

Summarize needs 

Brainstorm 

Phase 5: Resolution 



Evaluate ideas one at a time 
Consider workability 
Make sure agreements are specific 
Make sure agreements are balanced 
Sign agreement forms 



Comments? 



Phase 6: Departure 



Pill out evaluations and forms 
Explain follow-up 

Thank the participants and reinforce agreement 
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Sample 2 



Mediator Evaluation Form 



Participant: Evaluator: 

Circle Role Play: [List names of role plays being used for final evaluation] 

Use the scale indicated below to evaluate training participants in the areas specified. 

1 = Poor 2 = Needs Work 3 = Satisfactory 4 = Good 5 = Excellent 



Interpersonal Skills - Consider ability to: 

1 . communicate clearly and effectively to the disputants 1 2 3 4 5 

2. exhibit impartiality yet maintain control of the process 1 2 3 4 5 

Maturity - Consider your reaction to this person’s 

3. poise 

4. mannerisms 

5 . ability to make a professional impression 



Social Sensitivity - Consider the person’s sensitivity to and understanding of: 

6. the reactions and feelings of others 1 2 3 4 5 

7. their ability to respond appropriately and effectively 1 2 3 4 5 

Initiative - Consider: 

8. the ease and vigor with which this person approaches a new situation 1 2 3 4 5 

9. how the person carries the discussion to completion 1 2 3 4 5 

Cooperativeness 

10. The person’s attitude & ability to work with others, particularly as a co-mediator is... 1 2 3 4 5 



Mental and Verbal Ability - Consider: 

1 1 . the ease with which this person grasps new ideas 

12. the ease with which this person assimilates or rejects new discussion topics 

13. the person’s ability to express their thoughts in a clear and unbiased fashion 

Receptiveness to feedback - Consider the person’s response to: 

14. positive feedback 1 2 3 4 5 

15. constructive criticism 1 2 3 4 5 

16. Would you want this person to mediate a case for you? 1 2 3 4 5 

Other Comments? (consider: timing, issue clarification, assertiveness, productive vs. dictatorial 
guidance, whether present in moment vs. thinking ahead / lack of focus on speaker; Do we want to have 
participants formally evaluate each other? 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 



1 2 3 4 5 
1 2 3 4 5 
1 2 3 4 5 
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